Basics of Information Processing

I describe the basics of probability theory, statistical signal processing and information theory, and how these disciplines inter-relate to form the foundations of a theory of information processing. Examples are drawn from point-process applications. Johnson Basics of Information Processing 1 Probability theory Random variables can be either discrete or continuous valued. Discrete random variables can assume one of a countable set of values, and these values could be numeric (the integers, for example) or symbolic (color values). In either case, the probability of a discrete random variableX taking on the value x Pr[X = x] is denoted by PX(x). This probability function has the properties PX(x) 0 Px PX(x) = 1 An example of a discrete random variable is the Poisson, which is characterized by the probability function PX(x) = ( T )e T x! We use the shorthand notation X P( T ) to say that X is a Poisson random variable having parameter T . Continuous-valued random variables take on a continuum of values over some range. The probability that a continuous random variable X has a value somewhere in the interval [a; b] is given by